1. Field of the Invention
The present invention relates to an image processing apparatus, an image processing method, and a computer program. More particularly, the present invention relates to an image processing apparatus, an image processing method, and a computer program which generate a motion compensation (MC) image to be used for super-resolution processing, coding processing, etc., of an image.
2. Description of the Related Art
For a method of generating a high-resolution image from low-resolution images, super-resolution processing is known. The super-resolution processing is processing which obtains pixel values of individual pixels of a high-resolution image of one frame from a plurality of overlapping low-resolution images.
By super-resolution processing, it becomes possible to reconstruct an image having a resolution higher than a resolution of original images captured by an imaging device, such as a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), etc., for example. Specifically, super-resolution is used when generating a satellite photograph having a high resolution, and the like. In this regard, for super-resolution processing, for example, a description has been given by “Improving Resolution by Image Registration”, MICHAL IRANI AND SHMUEL PELEG, Department of Computer Science, The Hebrew University of Jerusalem, 91904 Jerusalem, Israel, Communicated by Rama Chellapa, Received Jun. 16, 199; accepted May 25, 1990.
A description will be given of a principle of super resolution with reference to FIGS. 1 and 2: Letters, a, b, c, d, e, f, shown in the upper part in FIGS. 1A and 1B indicate pixel values of a high-resolution image (SR (Super Resolution) image) to be obtained from low-resolution images (LR images) obtained by shooting a certain subject. That is to say, the letters indicate the pixel values of the individual pixels when the subject is represented by pixels with the same resolution as the resolution of the SR image.
For example, if it is assumed that the width of one pixel of an imaging device is twice the width of a pixel constituting a subject image, the subject image is difficult to be directly captured with the resolution. As shown in FIG. 1A, the left pixel, out of the three pixels of the imaging device, captures the pixel value of A, which combines the pixel values of a and b. The central pixel captures the pixel value of B, which combines the pixel values of c and d. Also, the right pixel captures the pixel value of C, which combines the pixel values of e and f. Here, A, B, and C indicate pixel values of pixels constituting an LR image obtained by shooting.
Together with the subject image in FIG. 1A, it is assumed that the subject image at a position shifted by a width of 0.5 pixel of the pixel constituting the subject is captured (captured while being shifted) by camera shake, etc., as shown in FIG. 1B on the basis of the position of the subject image in FIG. 1A. The left pixel, out of the three pixels of the imaging device, captures the pixel value of D, which combines the pixel values of a half of a, the whole of b, and a half of c. The central pixel captures the pixel value of E, which combines the pixel values of a half of c, the whole of d, and a half of e. Also, the right pixel captures the pixel value of F, which combines the pixel values of a half of e, and the whole of f. Here, D, E, and F indicate pixel values of pixels constituting an LR image obtained by shooting.
The following expression (Expression 1) is obtained from the shooting result of such LR images. By individually obtaining a, b, c, d, e, and f from Expression 1, it becomes possible to obtain an image having a higher resolution than the resolution of the imaging device.
                    [                  Expression          ⁢                                          ⁢          1                ]                                                                                  (                                                            1                                                  1                                                  0                                                  0                                                  0                                                  0                                                                              0                                                  0                                                  1                                                  1                                                  0                                                  0                                                                              0                                                  0                                                  0                                                  0                                                  1                                                  1                                                                                                  1                    /                    2                                                                    1                                                                      1                    /                    2                                                                    0                                                  0                                                  0                                                                              0                                                  0                                                                      1                    /                    2                                                                    1                                                                      1                    /                    2                                                                    0                                                                              0                                                  0                                                  0                                                  0                                                                      1                    /                    2                                                                    1                                                      )                    ⁢                      (                                                            a                                                                              b                                                                              c                                                                              d                                                                              e                                                                              f                                                      )                          =                  (                                                    A                                                                    B                                                                    C                                                                    D                                                                    E                                                                    F                                              )                                    (                  Expression          ⁢                                          ⁢          1                )            
A description will be given of super-resolution processing by back projection as a method of super-resolution processing, which has been known to date, with reference to FIG. 2. An image processing apparatus 1 shown in FIG. 2 is disposed, for example, in a digital camera, and performs processing on a still image obtained by shooting.
As shown in FIG. 2, the image processing apparatus 1 includes super-resolution processing sections 11a to 11c, a summation processing section 12, an addition processing section 13, and an SR image buffer 14. For example, LR0, which is an LR image obtained by shooting and has a low resolution, is input into the super-resolution processing section 11a, and LR1 is input into the super-resolution processing section 11b. Also, LR2 is input into the super-resolution processing section 11c. LR0 to LR2 are images captured consecutively, and have overlapping shooting ranges individually. When images are captured consecutively, in general, the ranges of a subject shot on shooting-result images are slightly shifted individually because of camera shake, etc., and thus they do not completely match, but overlap one another partly.
The super-resolution processing section la generates a difference image showing the difference thereof on the basis of the low-resolution image LR0 and a high-resolution image, an SR image, stored in the SR image buffer 14, and outputs a feedback value to the summation processing section 12. The feedback value indicates a value representing the difference image having the same resolution as the SR image.
In this regard, the SR-image buffer 14 stores an SR image, which is a high-resolution image generated by the super-resolution processing performed immediately before. When the processing has just started, and one frame of the SR image has not yet been generated, for example, an image obtained by up-sampling LR0 to an image having the same resolution as the SR image is stored in the SR-image buffer 14.
In the same manner, the super-resolution processing section 11b generates a difference image showing the difference thereof on the basis of the low-resolution image LR1 of the next frame and the high-resolution image, the SR image, stored in the SR image buffer 14, and outputs a feedback value representing the generated difference image to the summation processing section 12.
Further, the super-resolution processing section 11c generates a difference image showing the difference thereof on the basis of the next low-resolution image LR2 and the high-resolution image, the SR image, stored in the SR image buffer 14, and outputs a feedback value representing the generated difference image to the summation processing section 12.
The summation processing section 12 averages the feedback values supplied from the super-resolution processing sections 11a to 11c, and outputs an image having the same resolution as that of the SR image to the addition processing section 13. The addition processing section 13 adds the SR image stored in the SR image buffer 14 and the SR image supplied from the summation processing section 12, and outputs the SR image obtained by the addition. The output of the addition processing section 13 is supplied to the outside of the image processing apparatus 1, and is also supplied to the SR image buffer 14 to be stored.
FIG. 3 is a block diagram illustrating an example of the configuration of the super-resolution processing section 11 (the super-resolution processors 11a to 11c). As shown in FIG. 3, the super-resolution processing section 11 includes a motion-vector detection section 21, a motion-compensation processing section 22, a down-sampling processing section 23, an addition processing section 24, an up-sampling processing section 25, and a reverse-direction motion compensation processing section 26.
The high-resolution image, the SR image, which has been read from the SR image buffer 14, is input into the motion-vector detection section 21 and the motion-compensation processing section 22. The low-resolution image, LRn, obtained by shooting, is input into the motion-vector detection section 21 and the addition processing section 24.
The motion-vector detection section 21 detects a motion vector based on the SR image on the basis of the input high-resolution image, the SR image, and the low-resolution image, LRn, and outputs the detected motion vector to the motion-compensation processing section 22 and the reverse-direction motion compensation processing section 26. For example, by block matching between the SR image generated on the basis of a past captured image and the LRn image newly input, the motion-vector detection section 21 generates a vector indicating a movement destination of each block of the SR image in the newly input LRn image.
The motion-compensation processing section 22 performs motion compensation on the high-resolution image, the SR image, on the basis of the motion vector supplied from the motion-vector detection section 21 to generate a motion compensation (MC) image, and outputs the generated motion compensation image (MC image) to the down-sampling processing section 23. The motion compensation processing is processing which moves the pixel position of the SR image in accordance with the motion vector in order to generate a compensated SR image having a position corresponding to the LRn image newly input. That is to say, the motion-compensation processing section 22 moves the pixel position of the SR image to generate a motion compensation image (MC image) in which the position of an object on the SR image is matched with the position of the object on LRn.
The down-sampling processing section 23 generates an image having the same resolution as that of LRn by down sampling the image supplied from the motion-compensation processing section 22, and outputs the generated image to the addition processing section 24. Obtaining a motion vector from the SR image and LRn, and making the image obtained by performing compensation using the obtained motion vector to have the same resolution as that of the LR image is equivalent to performing simulation of the image obtained by shooting on the basis of the SR image stored in the SR image buffer 14.
The addition processing section 24 generates a difference image showing the difference between LRn and the image simulated in that manner, and outputs the generated difference image to the up-sampling processing section 25.
The up-sampling processing section 25 generates an image having the same resolution as that of the SR image by up sampling the difference image supplied from the addition processing section 24, and outputs the generated image to the reverse-direction motion compensation processing section 26. The reverse-direction motion compensation processing section 26 performs motion compensation in the reverse direction on the image supplied from the up-sampling processing section 25 on the basis of the motion vector supplied from the motion-vector detection section 21, and outputs a feedback value indicating the image obtained by performing the reverse-direction motion compensation to the summation processing section 12 shown in FIG. 2. A position of an object shown in the image obtained by performing the reverse-direction motion compensation becomes a near position to the position of the object shown in the SR image stored in the SR image buffer 14.